An image may represent a view of a scene as captured from the viewpoint of a camera. In some cases there may be more than one camera capturing different views of a scene. However, there will be some viewpoints of the scene which do not correspond to any of the camera viewpoints. The image may be a frame of a video sequence. Techniques such as Free-viewpoint video rendering (FWR) allow a novel view of a scene to be generated based on a set of multiple views of the scene from multiple camera viewpoints. The cameras are preferably calibrated and synchronized with each other so that the views of the scene can be combined correctly.
Based on the different views of the scene, a model (which may be referred to as a “geometric proxy”) of the scene may be constructed and a texture may be formed which can be applied to the model. The model, with the texture, can then be used to render the scene from a rendering viewpoint which may, or may not, be the same as one of the camera viewpoints. As well as recreating a “real-world” scene from a rendering viewpoint, the content of the real-world scene may be mixed with computer-generated content.
There are a number of issues which may need to be considered when generating a novel viewpoint of a scene. For example, relighting of the scene can be difficult. Textures extracted from images (e.g. frames of a video sequence) captured by cameras have implicit real-world lighting information, such that lighting artefacts are present (i.e. “baked-in”) in the textures.
One way of addressing the problem of how to relight the textures for a novel viewpoint is to control the lighting of the scene at the time when the cameras capture the different views of the scene. For example, diffuse lighting can be used in the initial video capture to avoid creating excess shaded areas that will damage the plausibility of the scenes rendered using extracted textures. The effects of lighting changes may be automatically addressed, but this may require an active lighting arrangement to deduce the material properties of the textures. However, relighting scenes with arbitrary lighting arrangements is considerably more challenging. Similar challenges apply to relighting textures with arbitrary lighting arrangements irrespective of how the textures were formed from captured images of scenes, for example when only one camera is used to capture an image of a scene from a single camera viewpoint and/or when the rendering viewpoint is the same as one of the camera viewpoints.